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This article describes the special equipment and test procedures 
developed for system-testing the No. 4 ESS toll system. It discusses 
novel methods developed for the coordination of hardware and software 
changes and testing in various system laboratory configurations. 
Planning, engineering, installation, and testing of early offices are 
described. Early field experience with the first operational No. 4 ESS 
is presented. This office, a selective routing tandem located in Chicago, 
went into service on January 17, 1976. 



I. INTRODUCTION 

A No. 4 Electronic Switching System was installed in Chicago to 
provide toll service between metropolitan Chicago and selected area 
codes in the United States. The new system was cut into service on 
January 17, 1976. During a three-month period, 150 local Chicago 
switching offices were connected through the new Chicago toll office to 
57 toll offices in California, Illinois, Florida, and Ohio. This cutover was 
the culmination of seven years of designing, manufacturing, and testing 
the new No. 4 ESS toll switching system. A second No. 4 ESS office in 
Kansas City, Missouri, was cut into service on July 3, 1976. Two more 
offices, Jacksonville, Florida, and Dallas, Texas, went into service in 
December 1976, and nine more offices are scheduled for service in 
1977. 

The development and testing of a toll system of this complexity was 
a sizable undertaking. The framework for integrating such a system was 
laid down in the initial planning stages of the project, and was supported 
throughout by the existence of design standards, documentation re- 
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quirements, and control procedures. Some of the important aspects, in 
this regard, in the No. 4 ESS development were: 

(i) Design standards for both hardware and software were developed 
and documented early in the project, and almost without exception, 
followed throughout. 

(ii) A procedure for the overall design of the system, as well as for each 
hardware and software unit, was specified. The requirements for No. 
4 ESS are based upon the requirements for a Bell System toll office. For 
each unit developed, two levels of design specifications were written, 
reviewed, and approved. The top level was known as a requirements 
design specification, and the second level as an implementation design 
specification. Each approved document became the vehicle for the next 
development step. 

(Hi) Procedures for documenting both hardware and software changes 
were developed very early in the project. These documents provided the 
vehicle for the institution of control and coordination procedures as 
required. 

(iv) The development of test facilities was begun at the same time 
as the development of the system and was subjected to the same stan- 
dards and development procedure requirements. 

(u) Early in the development, it was decided that there would be no 
field trial and that the development would progress directly from a 
laboratory test to the first commercial installation. By January 1970, 
Chicago had been identified as the site of the first installation and a first 
office planning committee had been formed. 

The following sections of this article describe how the overall labo- 
ratory and field testing was implemented. The excellent performance 
of the overall system at cutover resulted not only from good design but 
from the thorough planning, scheduling, and testing of the system before 
cutover. 

II. TESTING THE SYSTEM 

The testing and design verification of No. 4 ESS was accomplished in 
three distinct phases. In the first phase, the many units making up the 
system were tested individually. The definition of a "unit" was rather 
broad. For example, hardware unit testing included verification that 
individual circuit packs met their design requirements and also that all 
of the frame types used in No. 4 ESS worked together in a laboratory 
environment. Software unit testing encompassed everything from sim- 
ulating small software modules to verification that, for example, the data 
administration system correctly handled all specified input messages. 

The second phase of the job was called functional testing. In this phase, 
the objective was to verify that not only was each system function cor- 
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Fig. 1— Testing of No. 4 ESS. 

rectly performed, but that the system remained stable during the per- 
formance of any combination of instructions. 

The third phase was the evaluation and testing of the No. 4 ESS system 
under field conditions. In this phase, unit and functional tests similar 
to those run in the laboratory were repeated in order to provide a baseline 
from which to test the system in its operating environment. 

The testing of No. 4 ESS was a continuing effort for a period of five 
years, as shown on Fig. 1. While the objectives of each testing phase were 
distinct, the phases overlapped in time. This overlap was accomplished 
by beginning functional testing as soon as unit tests on a cohesive set of 
features were completed, and beginning field testing as soon as usable 
stability was achieved. Also shown on Fig. 1 are several major bench- 
marks referred to in the next sections. 
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III. UNIT TESTING 

3. 1 Hardware — early testing 

Initial verification of the hardware design was done using a logic 
simulator called LAMP, 1,2 running on a general-purpose computer. The 
LAMP simulator was capable of handling units varying in size from circuit 
packs of several hundred gates to frames of several thousand gates. In 
addition to verifying the hardware design, the test sequences used to 
detect and locate faults in the circuits were verified using LAMP. 

Once designed and LAMP-tested, circuit pack models were constructed 
in Bell Laboratories' model shops and tested by the designers. Some 
skeletonized frames were constructed at Bell Labs to verify concepts and 
check timing. However, the general reliance was on Western Electric for 
production of prototype equipment. The first of a kind for each frame 
was built by Western Electric and tested by the designers at Bell Labs 
using a minicomputer-controlled test set. Since the Bell Labs and 
Western Electric test sets were compatible, once the test data base was 
designed and verified at Bell Labs it could be transferred to Western 
Electric and all future frames tested in the factory. Exactly the same test 
data forms the basis for the Installation Test System described in Section 

5.2 and the generic diagnostic program. 3 As individual frame types were 
verified, groups of prototype frames were brought together in hardware 
laboratories where frame intercommunication and further operational 
checks were made. 

3.2 Software — early testing 

Just as the early hardware testing was done with a simulator (LAMP), 
early software was first verified on an ESS program simulator running 
on a general-purpose computer. While all software could not be com- 
pletely tested on the simulator, it proved to be a very efficient way of 
eliminating a large percentage of bugs early in the development. In all, 
90 percent of the generic code was tested on the simulator. As was 
mentioned previously, most of the diagnostic tests (also approximately 
90 percent) were verified in frame test and in the Installation Test Sys- 
tem. 

Before an operational 1A Processor was available, further testing of 
the software was accomplished using an available No. 1 ESS Processor. 
To do this, the No. 1 ESS Processor was slightly modified and connected 
via a No. 4 ESS peripheral unit bus branching frame to a signal processor, 
time-slot interchange, and time-multiplexed switch. The program, 
written in a combination of 1A Processor assembly language and ESS 
Programming Language (epl), was translated so as to run on the No. 
1 ESS Processor. Sufficient progress was made in checkout of the oper- 
ating system and call-handling programs to allow a call to be completed 
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on November 2, 1972, seven months before a 1 A Processor was available 
for general-program debugging. 

3.3 System laboratories 

Two system laboratories were devoted to testing No. 4 ESS. A system 
laboratory consists of a 1 A Processor, a minimum set of No. 4 ESS pe- 
ripheral hardware, and transmission switching interface and toll terminal 
equipment. An additional system laboratory containing only a 1 A Pro- 
cessor was also used, mainly for testing 1A Processor common pro- 
grams. 4 - 5 There were two reasons for having two complete system labo- 
ratories. First, the initial laboratory, which we called Delta, contained 
prototype equipment, while the second laboratory, which we called Echo, 
was made up of production equipment. Having Echo guaranteed that 
testing, especially the testing of changes, was carried out in a realistic 
environment. The second reason for two system laboratories was to 
double the available program testing time in order to meet the cutover 
schedule. 

As shown on Fig. 2, activity in each system laboratory began with in- 
stallation and testing of the equipment and utility system. 6 Program 
checkout began on June 1, 1973, in Delta and on April 1, 1974, in Echo. 
However, in both system laboratories, the initial program debugging 
began in an environment of continuing hardware debugging and hard- 
ware change activity. Many of the hardware changes, involving both 
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wiring and circuit pack change, came about as a result of problems found 
when operating with the generic program. One measure of change ac- 
tivity is shown in Fig. 3. The curve shows the cumulative number of wires 
changed in the two system laboratories during 1974 and 1975. Change 
activity consumed a great deal of laboratory time in early 1974, but began 
to level off by August when basic operating stability was achieved. It then 
increased slightly and continued at a relatively uniform rate throughout 
most of 1975. However, in this later period the changes generally were 
such that they did not cause extensive interference with program de- 
bugging. The approximately 40,000 backplane wires changed in the 
system laboratories in 1974 and 1975 were about 1.5 percent of the total 
number in the laboratories. 

Beginning August 1, 1974, both system laboratories were devoted to 
full-scale program checkout. Full-scale program checkout was defined 
to be the point where 16 hours a day were scheduled for program de- 
bugging and where at least 80 percent of the scheduled time was actually 
productive. Unproductive time was caused by such things as hardware 
failures or change activities not being completed on schedule. 

From the time program debugging began, the system laboratories were 
operated in two different modes. One was a closed shop, batch-type mode 
where programmers submitted test decks, which were run by operators, 
and printouts returned. Of the total system laboratory time available 
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Fig. 3 — Cumulative backplane wiring changes in Delta and Echo laboratories. 
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for program debugging during the unit testing interval, approximately 
55 percent was devoted to batch operation. The availability of a powerful 
utility system and automated testing aids 6 made this mode of operation 

viable. 

The remaining system laboratory program debugging time was de- 
voted to hands-on operation. Hands-on time was assigned when required 
by specific problems, such as those whose solution required the interplay 
of a programmer with debugging tools and a hardware designer with 
oscilloscope, and also where problems existed which were impeding fu- 
ture debugging progress in other areas. In this second situation, a small 
group of programmers was assigned blocks of time. Much of the time was 
used running "instant turn-around" batch-type test jobs for these pro- 
grammers. 

Using both batch and hands-on techniques, a program debugging rate 
of about 20,000 instructions a month was sustained from September 1973 
through February 1975. The fast startup of program debugging, shown 
in Fig. 4, was a result of the extensive pretesting of both the hardware 
and software. 

IV. FUNCTIONAL TESTING 
4. 1 Laboratory system testing 

An independent system test group was chartered to provide a systemic 
testing viewpoint. A set of system -level test sequences was generated 
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and executed in a controlled, highly interactive environment. A subset 
of these test sequences was used as a generic system laboratory accep- 
tance test. This acceptance test was executed periodically during the 
functional testing interval, and the statistical performance results, as 
a function of time, were used to indicate the rate of achieving system 
stability. The results of the acceptance tests are presented in Figs. 5 and 
6. Figure 5 shows the percentage of tests in an acceptance test that exe- 
cuted as expected. Figure 6 shows the mean time between unscheduled 
major recovery events. Both are graphed as functions of time. Absolute 
numbers were not used as prime indicators, rather the slopes were used 
and, as can be seen, both figures indicate that system stability was being 
achieved. 

The highest-priority goal of the functional testing interval was to 
achieve system stability in the presence of system faults. This was fo- 
cused on with the firm conviction that this characteristic was funda- 
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Fig. 6 — Mean time between major recovery events. 

mental to our ability to resolve problems in an operational office. This 
important goal was achieved. 

The key point was that this was an independent effort. The people 
were responsible for the system, not for any individual component of it. 
They ensured that the system was ready for release, using the system 
laboratory as the testing vehicle. This was achieved by the system test 
group being the focal point of change control and also the primary in- 
terface with the field test group. 

4.2 Laboratory System Evaluation 

A great deal of effort was spent to ensure that No. 4 ESS would meet 
all of its real-time objectives. Early in the project's life, numerous real- 
time simulations were developed and executed to establish confidence 
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that these objectives would be met. These simulators were designed such 
that after the system laboratories became operational, the same exper- 
iments could be run there to verify the simulator results. These experi- 
ments were done in three areas: (i) system response versus call attempts 
per hour; (ii) system overload control responses as a function of call load; 
(Hi) system engineering requirements versus real-time capacity. In all 
three cases, the simulator results were verified, and in the case of real- 
time capacity, allowed us to increase the advertised capacity from 
350,000 to 550,000 engineered call attempts an hour. 

In summary, we were able to verify our simulated system performance 
characteristics, and thus, to achieve all engineering requirement system 
objectives. 

V. ENGINEERING AND PLANNING EARLY OFFICES 
5. 1 Selection of Early Offices 

There were many locations at which there was need for a large toll 
switching machine. Chicago was chosen jointly by AT&T, Bell Labs, and 
Illinois Bell as the site for the first No. 4 ESS for a number of reasons. It 
is conveniently close to both Bell Labs at Indian Hill, the development 
location, and to Western Electric at Lisle, the principal manufacturing 
location. This proximity facilitated support during the installation, 
testing, and initial service periods. In addition, the office, called Chicago 
7, was planned to be a new-start office rather than a replacement. This 
avoided many problems that would have been associated with the 
cutover of in-service facilities from one office to another. It permitted 
the development of a simple phased cutover and contingency plan. 

Kansas City was chosen as the site of the second office, called Kansas 
City 2, in part, in order to face the problems associated with replacing 
an in-service 4A crossbar office with a No. 4 ESS. These problems include 
the reuse of existing toll terminal equipment and the development of 
flash cutover procedures. 

The selection of subsequent offices was done by Long Lines and AT&T 
and was based on need for additional switching capacity and the buildup 
in Western Electric production rate. 

For both Chicago 7 and Kansas City 2, office planning committees 
were organized with representatives from AT&T Headquarters, Long 
Lines Headquarters, local Long Lines Operations and Engineering, 
Western Electric Product Engineering Control Center and Installation, 
and Bell Labs. These committees were distinct from and in addition to 
the normal operating company cutover committees. The planning 
committees established schedules for office engineering, manufacturing, 
installation, and testing. Progress was reviewed regularly and problems 
were solved as they arose. Because many decisions were required before 
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standard methods had been developed and documented, Bell Labs en- 
gineers did much of the engineering of the processor and network for 
Chicago 7. Western Electric's Product Engineering Control Center 
participated in this effort and developed the standard engineering 
questionnaire as a result. The transmission facilities for Chicago 7 were 
all new and posed no special problems. They were engineered by Long 
Lines and installed by Western Electric. Because Kansas City 2 was to 
be a replacement for an existing 4A crossbar office, the reuse of an ex- 
isting set of transmission facilities raised many questions about methods 
of connection to No. 4 ESS, trunk circuit compatibility, and segregation 
of private line and switched circuits that were mixed in existing carrier 
systems. The economics of reusing existing older equipment compared 
with replacement by new toll terminal equipment was also considered. 
Bell Labs System Engineering worked closely with operating company 
engineers to answer these questions. 

As part of the system development, standard floor plans were devel- 
oped for the major elements of the system, i.e., 1A Processor, time-di- 
vision network, signal processor-voiceband interface cores, and toll 
terminal equipment. Therefore, the preparation of floor plans for Chi- 
cago 7 and Kansas City 2 was simplified to arranging these large blocks, 
rather than individual frames, in the available space. 

5.2 Installation Test System 

Concurrent with the development of the No. 4 ESS, Western Electric 
engineers developed factory frame tests and installation tests based on 
the common tests described earlier. A key element in the design of No. 
4 ESS is that a single set of tests be developed and used for all testing of 
a particular type of unit, i.e., single unit testing in the factory, system 
testing during installation, and diagnostic testing in an in-service office. 
Installation Test System (ITS) is the name of the program used by the 
Western Electric installation testers to control the execution of unit and 
subsystem tests. ITS is characterized by permitting tests to be run si- 
multaneously on several different pieces of equipment. A prerequisite 
for the beginning of system testing at Chicago 7 was that all ITS unit and 
subsystem tests pass successfully. The testing activity at Chicago 7 
spanned a period of 1% years starting in mid- 1974 when ITS tests were 
begun. Figure 7 shows the sequence of tests through the phased cutover 
in early 1976. 

5.3 Field Test Plan 

A field test plan consisting of functional tests, environmental tests, 
and an overall acceptance test was written and used as a final check that 
the system met its functional objectives at specified environmental 
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limits. About eleven thousand individual functional tests were written 
and most were executed several times during the system test interval 
at Chicago 7. The environmental tests verified system performance at 
the limits of high temperature and low voltage. A program-generated 
call load of 80,000 calls per hour was applied in order to verify the ade- 
quacy of the office engineering. In addition, six persons at real telephones 
placed several thousand calls in order to get a subjective measure of the 
system performance. Final acceptance of the system as ready for com- 
mercial service was based on tests during which the system was operated 
for several days by the Long Lines craft while it was switching test 
traffic. 

This functional testing interval began on one shift a day in November 
1974, increased to two shifts a day in January 1975, and went to three 
shifts a day in March of 1975. During this period, many tasks competed 
for system time. In addition to the execution of field test plans, time was 
required for the installation of hardware design changes, program cor- 
rections, ordinary maintenance troubleshooting, and testing and 
alignment of the connecting trunks by Long Lines. Although several 
tasks might be executed at the same time, careful scheduling was re- 
quired to avoid the many incompatible combinations of system use. From 
the earliest stages of planning the system integration and testing, it was 
agreed that there would be no scheduled program debugging at Chicago 
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7 in order to avoid yet another demand for system time. Despite thorough 
verification in the system laboratories, there were occasions when pro- 
gram tests were run at Chicago 7 to identify problems that were unique 
to the office because of its size, hardware configuration or data base. 
Despite these occasional lapses, the normal procedure was for system 
performance problems to flow from the field test group to the system 
test group and for integrated programs and tested changes to flow to 
Chicago 7. 

VI. CHICAGO 7 PERFORMANCE EVALUATION 

The cutover of Chicago 7 was accomplished on the schedule set in 1972. 
The cutover was of the phased type, covering the period January 17 to 
April 24, 1976, during which about 4500 toll connecting trunks from 150 
local offices in the 312 NPA and a nearly equal number of intertoll trunks 
to 57 toll offices in foreign NPAs were put in service. Traffic data for 
Chicago 7 are shown in Fig. 8. Traffic built up as trunks were added and 
reached a busy-day high of 350,000 and a busy-hour of 31,000 calls at the 
completion of the phased cutover. Additional trunks to the NPAs origi- 
nally served were added in the summer of 1976 and trunks to New York 
City were added starting in the fourth quarter of 1976. The trunking 
configuration for Chicago 7 at the start of 1977 is shown in Fig. 9. 

There are several criteria for judging the performance of an electronic 
switching system. These include the number of ineffective attempts, as 
a measure of the grade of service received by the customer, and measures 
of the required maintenance effort on the part of the operating company. 



SYSTEM INTEGRATION 1291 



SELECTIVE ROUTING TANDEM 



LOCAL 
CHICAGO 

OFFICES 



NEW YORK 




ORLANDO 



57 TOLL OFFICES 
150 LOCAL OFFICES 
9,000 TRUNKS 



Fig. 9 — Chicago 7 trunking. 



Prior to cutover, two criteria were set for each of a number of measure- 
ments, Table I. These were a desirable objective and second a level at 
which the advisability of putting the office into service would be ques- 
tioned. 

Ineffective attempts result from sources external to the switching 
machine as well as from internal causes. Figure 10 shows the history of 
ineffective attempts at Chicago 7 for the first year of service. The most 
recent data show total ineffective attempts averaging less than 1 per- 
cent. 

The replacement rate for plug-in apparatus is one measure of the 
maintenance effort required. Figure 11 shows the monthly average of 
the daily replacement rate through the first year of service. During the 
first 122 days of service, there were 413 plug-ins of all kinds replaced, 
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for an average replacement rate of 3.4 a day. The present average re- 
placement rate is between 2.5 and 3.0 a day. 

A maintenance interrupt occurs whenever the system experiences an 
event of sufficient importance to interrupt the normal flow of program 
control in order to take immediate action. Maintenance interrupt actions 
require only a few milliseconds and have no impact on customer service. 
They are, however, a measure of required maintenance activity. Another 
measure of system performance is the number of occurrences for which 
the system response was a memory reinitialization phase 3 of sufficient 
severity to have an impact on call processing. Neither a phase 2 nor a 
phase 3, either of which can be run automatically, will disturb calls in 
the talking state, but these phases will prevent the completion of calls 
in the process of being established. A phase 4, the most severe reiniti- 
alization, can be induced only by manual action. A phase 4, in addition 
to preventing completion of transient calls, will disconnect stable calls 
in the talking state. One observation is that there is a tendency for one 
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stimulus, e.g., a hardware failure, to trigger a number of phases in rapid 
succession. Each such incident has been studied in detail and these 
studies have resulted in a number of improvements in the maintenance 
program strategy. Transient hardware errors resulting from marginal 
equipment failures are one of the most difficult classes of problem for 
the system to cope with. Figure 12 shows both the monthly average of 
the number of interrupts a day and the occurrences of maintenance in- 
cidents that required one or more reinitialization phases. 

VII. SUMMARY 

The successful introduction of No. 4 ESS into commercial service in 
Chicago 7 resulted not only from good design but also from a large effort 
directed toward system testing and integration. Initiated early in the 
development process, these efforts included careful planning for mea- 
suring system performance and coordination of the introduction of 
hardware and program changes. This ensured that the process converged 
toward a stable and satisfactory system design. 

The overall performance of the Chicago 7 system has been excellent. 
No fundamental system problems have been encountered, although some 
software and hardware improvements have been incorporated as a result 
of operational experience. It is expected that all performance objectives 
will be reached in the near future as a result of additional craft training 
and recently introduced system improvements. 
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